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ABSTRACT 


4 The selection process of proposals is a crucial component of scientific progress and innova- 

5 tions. Limited resources must be allocated in the most effective way to maximise advance- 

6 ments and the production of new knowledge, especially as it is becoming increasingly clear 

7 that technological and scientihc innovation and creativity is an instrument of economic pol- 

8 icy and social development. The traditional approach based on merit evaluation by experts 

9 has been the preferred method, but there is an issue regarding to what extent such a method 

10 can also be an instrument of effective policy. This paper discuss some of the basic processes 

11 involved in the evaluation and selection of proposals, indicating some criterion for an optimal 

12 solution.. 
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1. Introduction 


The selection process of scientific activities is nsnally performed by review of the merit 
and originality of the proposal. It is generally thonght that such a competitive process will 
guarantee the emergence of the projects with the higher potential to offer new insights and 
produce more innovation. However, research is a very uncertain business and the ex-ante 
innovation potential may not be realised at all or achieved only partially. The evaluation 
procedures are obviously measuring only the potential of innovation and scientihc advance 
and therefore are by dehnition affected by errors. Such errors are judgemental, cultural, 
numerical or simply social as they result from different school of thought or various factions 
in the scientihc community. It is reasonable to ask if there is a ’’best” strategy in the shaping 
of a research program to yield an optimal level of selection for the proposal that would take 
into account these uncertainties. 

A naive approach would simply use only the very top projects, but this choice would 
result in a reduced diversihcation of approaches and methods, thereby increasing the risk. 
On the other hand, accepting all projects will guarantee the maximum innovation, but it 
will be wasteful of resources and morally unacceptable because there will be no incentive to 
produce, sound, well-based proposals. 

Detailed analysis of project selection including asymmetric informations and outside 
choices have been performed ( Bar and Gordon[] 2014 )using highly sophisticated mathematical 
methods, but this short paper proposes to analyse this mechanism via a simple model of 
evaluation and project distribution, resulting in an understanding of the underlying mech¬ 


anism. The approach allows to estimates some optimal thresholds for maximising scientihc 
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results and innovation. 


2. The model 


The various functions and distributions in this paper will be described in terms of the 
evaluation value, x, that is the score obtained via an ex-ante evaluation by a certain proposal, 
project or other forms of scientihc documents as a result of a solicitation or a call for tender. 

The detailed evaluation procedure is not important here, but only that whatever score 
is used it must be a monotonic function of the implicit ’’value” of the proposal. This is not 
such a strong restriction since any indicator of value should realise a consistent ranking of 
proposals. 

The density of proposals as a function of the score will be denoted by p{x) so that the 
number of proposal up to a certain score A will be given by 

N{X) = p{x)dx (1) 

normalized by the total number of proposals. 


Ni= p{x) dx. 


Jo 

The distribution of projects with respect the score is very asymmetric. The estimated 
probability density obtained from real evaluation exercises (FigQ show that evaluations tend 
to cluster at a larger value than the average grade, with very small tails at high and low 
levels. The data here has been obtained from past evaluation results of the FP7 EU program 
(Barbante, 2015, Pers. Comm.) . Few projects score the maximum and few projects are 
indeed so bad to deserve the minimum score, as a result the distribution has a internal peaks. 
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We can also describe the innovation content of a project with an innovation density v(x) , 


meaning that v(x) is the innovation content of a proposal with score x. The total innovation 
of the proposals np a certain score is then given by the integral 



52 where a normalisation has been introdnced so that the maximnm valne of innovation achiev- 

53 able with this particnlar set of proposals is one. 

The innovation content is an abstract qnantity that indicate the amonnt of new resnlts, 
advancements or in general new science attained by the project. It is difhcnlt to model snch 
fnnction, bnt it seems that in order to have the evalnation process make any sense at all it 
mnst be a growing fnnction of the score x, possibly a very nonlinear fnnction as we expect 
that the best project will have a considerably larger potential than the rest. In this case we 
can choose a simple behavionr: 


v{x) = exp(aa;) — 1 


54 the scale a will give ns the strength of increase with the increasing score. Figj^ shows an 

55 example of snch a density for a = 10. 

The shape of the project density fnnction snggests that we can model it with a simple 
fnnction 
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where xq is the score at the peak of the distribntion and a is the width of the distribntion. 


57 


Figj^ shows two examples together of the project density fnnction and of the innovation 


58 density fnnction. 
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3. The optimal choice 


A typical procedure would proceed to accept proposals starting from the maximum score 
working down the list toward lower values, usually until funds are exhausted. The issue we 
would like to investigate is if there is a way to determine a theoretical optimal choice (in 
some sense) to choose the funding threshold , A, such that proposals scoring higher than that 
will be retained and the others declined. We can dehne the problem as follows: we need to 
hnd the cut A that gives the maximum total innovation /(A) with the minimum number of 
proposals. The total innovation for the proposal retained above the cut A is therefore given 
by 


'^i(A) = / p{x)v{x)dx 


and the number of retained proposal is 


1 

Nr{X) = 

60 we would like to get the maximum innovation with the minimum of proposals, so the desired 

61 threshold A is such that max{Ji) and min{N^. We can observe however that the minimum 

62 of retained proposal is equivalent to maximising the number of declined proposals, so we can 

63 use the total number of declined proposals 


'^ 2 (A) = p{x)dx 


such that 


= min ( — / p{x) dxj = max ( — / p{x) dx ) = max( J 2 ) 

.X/i J\ J Jo 
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and since Ji and J 2 are positive, we can look for the maximnm of a cost fnnction J 

/•A 

x)v(x) dx+ p{x) dx 

Jo 

Figj^ shows the behavionr of these fnnctions. The total innovation is large when all 
projects are retained and becanse of the steep behavionr of the innovation density fnnction 
the innovation is really all concentrated in the best projects, i.e. those scoring 0.7 and 
higher, as it is expected. The nnmber of projects as a fnnction of the threshold drops less 
rapidly as a conseqnence of the internal niaximnm of the density fnnction. As a resnlt the 
cost fnnction has also an internal maximnm. For the case in the pictnre the maximnm total 
innovation with the minimnm nnmber of projects can be obtained with a score threshold of 
0.74, corresponding to retaining 26% of the projects. 



4. Sensitivity Tests 

Different behavionr of the innovation fnnctions will hx the maximnm innovation obtain¬ 
able from a given set of proposals, becanse the innovation is basically the overlap integral 
between the projects distribntion and the innovation cnrve. Figj^ shows the innovation for 
varions valnes of the scale factor. As the innovation gets more concentrated towards the 
higher valnes of the score fewer and fewer project will contribnte to the total innovation. 
Conversely, assnming a weaker dependence of the innovation on the score reqnires more 
projects to contribnte to the totale innovation. 

By there same token, Figj^ shows what happened for varions project density fnnctions. 
If the project density distribntion is peaked at low valnes there will be a limited overlap and 
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a small total innovation is generated, whereas a distribution peaked toward high values will 
result in a large total innovation. 

We can get higher thresholds if we assume a faster increase of the innovation density with 
the score. Table [2] shows some results with different scales in the innovation function. The 
scale gives a measure of the gap between the best and the worst proposal. It is interesting 
to note that it is very difficult to get large rejection values, above 95%, corresponding to 
single digits success rates. This is quite reasonable for innovation that differs by orders 
of magnitude, but it is sub-optimal if there is a more uniform innovation values of the 
projects. It appears that for more moderate innovation densities values around 30% are more 
reasonable. Interestingly, these numbers corresponds to experimentally obtained values from 


the National Science Foundation (NSF 2014). In the period 2004-2013 the overall success 


rates for proposals, defined as the ratio between the proposal funded with respect to the 
total number of proposal presented, has always been higher than 20%, peaking at 32% in 
2009, decreasing to 22 — 24% in the following years. Disciplinary differences can be seen, 
GeoSciences peaked in 2009 at 45%, in the same year Social and Behavioural and Economics 
sciences reached 30% and Mathematics and Physics 40%. In Europe, the FP7 cooperation 
program yields a general success rate of 17%, averaged over the five years of the program 


and the environmental program in particular yielded a similar result (Helming et ah 2014) 


Statistics for the grants from the European Research Council are much lower. Success rates 
for Starting Grant from 2007 to 2014 were around 10%, Advanced Grant fared a little better 
around 13% (ERG Webpage, 2014). 


7 








103 


5. Conclusion 


104 This simple model indicates that the selection process will yield optimal impacts only 

105 if the realistic distribution of the innovation potential is considered. The example from a 

106 realistic evaluation shows that the innovation potential, as it is measured by the score, is 

107 distributed into a larger portion of initiatives and ideas - i.e. proposals - that simply those 

108 scoring at the maximum level. As a consequence, optimal innovation can be reached only by 

109 accepting a wider range of proposals. Imposing very high thresholds, like in the ERC case, 

110 will have a sub-optimal impact. 

111 The model is a very simplified analysis that is of course missing cost considerations and 

112 more generally policy constraints and decisions that can affect the general mechanisms out- 

113 line here, but the main issue of having a mismatch of the value of the proposals and the 

114 implicit innovation potential looks fundamental. Unless, of course, we admit that the evalu- 

115 ation process is inaccurate and the scores do not reflect the innovation potential. Probably 

116 a scoring system that is not expressed in terms of continuous values, but is designed with 

117 categories as Certainly Reject, Certainly Accept, Accept with Reserve, for instance, would 

118 be less mechanistic and more in line with the ultimate policy goal to maximise innovation 

119 and production of knowledge. It is clear that a system with very low values of acceptance 

120 resembles a lottery system and though it is right to have an award system in place to rec- 

121 ognize exceptional achievements, it is doubtful it can be an effective instrument of policy. 

122 Nobel prizes can come after the research is done, but they are not the way to stimulate and 

123 encourage new research. 
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project density fnnction. The innovation scale nsed here has a = 10. A 
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projects of low innovation potential therefore only a small fraction are worth 
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retained and higher innovation. [12] 
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Center 

Threshold 

Retained Projects 

Innovation Obtained 

0.2 

46% 

15% 

0.85 

0.4 

61% 

17% 

0.85 

0.6 

74% 

23% 

0.83 

0.8 

83% 

32% 

0.80 


Table 1. Optimal thresholds, project retained and normalised total innovation achieved 
by retained projects for various values of the position of the center of the project density 
function. The innovation scale used here has a = 10. A distribution shifted very much 
toward low values indicates a set of proposal projects of low innovation potential therefore 
only a small fraction are worth retaining to achieve the nest result. In this case the total 
(unnormalized) innovation value is much less. 
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Scale a 

Threshold 

Retained Projects 

Innovation Obtained 

1 

68 % 

49% 

0.64 

5 

74% 

37% 

0.71 

10 

79% 

27% 

0.82 

20 

85% 

17% 

0.91 


Table 2. Optimal thresholds, project retained and normalised total innovation achieved 
by retained projects for various values of the scale of the innovation density function. The 
project density function has been kept hxed at xq = 0.7, a = 0.1. A steeper behaviour od 
the innovation function results in fewer projects retained and higher innovation. 
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Fig. 1. The estimated distribution of scores from an FP7 evaluation exercise, superposed 
to a theoretical innovation density curve. 
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Fig. 2. The project density function and the innovation density functions, using xq 
0.6, cr = 0.1 and a = 10 
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Fig. 3. The total number of declined proposals, J 2 , the total innovation achieved, Ji and 
the cost function J as a function of the threshold value A. The case shown here is for a 
project density function with xq = 0.6, a = 0.1 and for an innovation density function with 
a scale a = 10. 
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Fig. 4. Innovation density functions for a = 1,5,10, 20. A project density function (dashed 
line) using xq = 0.6, a = 0.1 is also shown for reference 
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Fig. 5. Project density functions using xq = 0.2,0.4, 0.6, 0.8, a = 0.1. An innovation density 
with a = 10 (dashed line) is also shown for reference 
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